AN INVESTIGATION OF DISTRIBUTED COMMUNICA- 
TIONS SYSTEMS AND THEIR POTENTIAL APPLICA- 
TIONS TO THE COMMAND AND CONTROL STRUCTURE 
OF THE MARINE CORPS 



By Edmund Andrew Lucke 



NAVAL POSTGRADUATE SCHOOL 

Monterey, California 







AN INVESTIGATION OF DISTRIBUTED COMMUNICATIONS 
SYSTEMS AND THEIR POTENTIAL APPLICATIONS TO 
COMMAND AND CONTROL STRUCTURE OF THE 
MARINE CORPS 

by 

Edmund Andrew Lucke 
December 1979 



Thesis Advisor: J. M. Wozencraft 



Approved for public release; distribution unlimited 



T 19 




SECURITY CLASSIFICATION of THIS PACE (When Dmtm Entered) 



REPORT DOCUMENTATION PAGE 


READ INSTRUCTIONS 
BEFORE COMPLETING FORM 


1. REPORT NUMBER 


2. GOVT ACCESSION NO. 


3. RECIPIENT’S CATALOG NUMBER 


4. TITLE (end Submit) 

An Investigation of Distributed Communi- 
cations Systems and Their Potential 

Applications to the Command and Control 
Structure of the Marine Corns 


S. TYRE OF REPORT * PERIOD COVEREO 

Master's Thesis; 
December 1979 


• . PERFORMING ORG. REPORT NUMBER 


7. AUTHOR!'*,) 

Edmund Andrew Lucke 


S. CONTRACT OR GRANT NUMB ERf •) 


t. performing organization name ano adores* 

Naval Postgraduate School 
Monterey, California 93940 


10. PROGRAM ELEMENT. PROJECT, TASK 
AREA A WORK UNIT NUMBERS 


II. CONTROLLING OFFICE NAME ANO AOORESS 

Naval Postgraduate School 
Monterey, California 93940 


12. REPORT OATE 

December 1979 


12. NUMBER OF PAGES 

83 


14. MONITORING AGENCY NAME * AOORESSfJ/ different from Controlling Office) 

Naval Postgraduate School 
Monterey, California 93940 


IS. SECURITY CLASS, (ml 'him ripen) 

Unclassified 


1S«. OECLASSI FI CATION/ DOWNGRADING 
SCHEDULE 


1 l«. DISTRIBUTION STATEMENT (o 1 'him Ktpon) 

Approved for public release; distribution unlimited 


17. DISTRIBUTION STATEMENT (ot thm mmettmct entered in Block 20, it different from Report) 


It. SUPPLEMENTARY NOTES 


It. KEY WORDS (Continue on reeeree elde it nmcememrr end Identity by block number) 

Distributed Communications, Optimal Routing, Network 
Management, Short Path Algorithms, Routing Protocols, 
Network Synchronization 


20. ABSTRACT (Continue on reeeree elde It neceeemy end Identity by block mmtber) 

The Marine Corps Tactical Command and Control System 
(MTACCS) is expected to provide increased decision making speed 
and power through automated processing and display of data which 
previously was processed manually. The landing Force Organiza- 
tional Systems Study (LFOSS) has challenged Marines to examine 
"how Command and Control will be exercised, and what the 
organizational impact will be" of MTACCS. (MAR 78) 



DD 1 ja" M 73 1473 EDITION OF 1 NOV «S IS OBSOLETE 
(Page 1) S/N 0102-014- 6601 | 



SECURITY CLASSIFICATION OF THIS P AO E (When Dmtm Entered) 



f#cuwry CLAIHFICATIOW O W Tn|| frmtm *«<•*« tf 



This thesis is aimed primarily at the post MTACCS concepts 
of Command, Control and Communications (C3) . It has been 
heavily influenced by the areas of concern which the acquisi- 
tion of MTACCS has generated. The areas of mobility, maintain- 
ability, survivability and implementation are all areas in 
which distributed systems hold promise of improvement upon those 
provided by the hierarchically structured MTACCS architecture. 

Here we examine the required architecture for a second 
generation automated C3 system entitled the Mobile Command 
Concept (MCC) . 



DD Form 1473 




2 SICyHlTV CL AMlPlC ATJOW 0^ TWI» 



Approved for public release; distribution unlimited 



An Investigation of Distributed Communications Systems 
and Their Potential Applications to the 
Command and Control Structure of the Marine Corps 



by 



Edmund Andrew Lucke 
Captain, United States Marine Corps 
B.S., United States Naval Academy, 1970 



Submitted in partial fulfillment of the 
requirements for the degree of 



MASTER OF SCIENCE IN ELECTRICAL ENGINEERING 



from the 



NAVAL POSTGRADUATE SCHOOL 
December 1979 



* :> 

J ' 

c . \ 






ABSTRACT 
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The Marine Corps Tactical Command and Control System 
(MTACCS) is expected to provide increased decision making 
speed and power through automated processing and display of 
data which previously was processed manually. The Landing 
Force Organizational Systems Study (LFOSS) has challenged 
Marines to examine "how Command and Control will be exer- 
cised, and what the organizational impact will be" of 
MTACCS. (MAR 78) 

This thesis is aimed primarily at the post MTACCS 
concepts of Command, Control and Communications (C3) . It 
has been heavily influenced by the areas of concern which 
the acquisition of MTACCS has generated. The areas of 
mobility, maintainability, survivability and implementation 
are all areas in which distributed systems hold promise of 
improvement upon those provided by the hierarchically 
structured MTACCS architecture. 

Here we examine the required architecture for a second 
generation automated C3 system entitled the Mobile Command 
Concept (MCC) . 
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I. 



INTRODUCTION 



A. BACKGROUND 

Modern warfare has become so intense and so lethal that 
correct and timely decisions by the Commander are essential 
to victory. The necessity of making a large number of such 
decisions over an extended period of time appears to accu- 
rately describe the nature of the problem facing the Com- 
mander and his staff in many future conflicts. Automated 
aids to the decision-making process have been proposed as 
the best way to help the Commander achieve his stated mission. 
A large number of such systems will soon be arriving in the 
inventory of the Armed Forces of the United States. They 
represent the solution to the problem of how to transmit, 
process and display increasing volumes of digital information 
rapidly, accurately and securely within the traditional 
hierarchical Command and Control structure. This was a 
natural first approach to the problem: automation of routine 

manual functions. However, as large and more sophisticated 
systems came upon the scene to "aid" the Commander, it has 
been recognized that they may instead frustrate his success 
if they fail to perform as expected. It would be even more 
galling if they were to fail because of inability on our 
part to effectively operate and properly maintain them, or 
if the C3 structure which they support is not flexible enough 
to adapt to tactical needs. Unstated in these concerns is 
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the very real problem that these large and more capable 
systems may not be able to withstand the rigorous environ- 
ment in which they will have to operate. Indeed their 
presence may make a Command Post an even higher value 
target. (NAV 78) 

The complex set of problems surrounding C3 support to 
tactical commanders became more vexing when materials were 
published outlining a Soviet counter-C3 strategy. Radio 
Electronic Combat (REC) is their plan for integrating signals 
Intelligence (SIGINT) , target acquisition, Electronic 
Countermeasures (ECM) , Electronic Support Measures (ESM) , 
electronically supported firepower assets, and Electronic 
Counter-Counter Measures (ECCM) . The purpose of this inte- 
gration is a planned sequence of activities that will 
selectively deprive Soviet opponents of control of the 
tactical electromagnetic environment. 

REC has two basic tenets. First, it attempts to locate 
communication "keystones" upon which the command and control 
of U.S. tactical forces and weapons systems are dependent. 
Second, as knowledge of these "keystones" is developed by 
Soviet intelligence they are prioritized according to their 
expected impact on the battle. This is done so they may be 
selectively neutralized (by ECM or supporting arms) . Such a 
strategy demands a move away from rigidly structured network 
architectures. (MAR 79) Chapter III discusses the implica- 
tions of structured networks. 

It is not the nature of technology to be a panacea; 
rather, the ingenious application of technology is the real 
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strength it brings to a problem. Thus at the heart of the 
long term C3 problem for the Commander is the need to find 
an ingenious application of the rapidly expanding technology 
in this area. Merely repeating previous solutions with more 
capable devices is not the answer, since the structure of 
these solutions themselves appears to be their greatest 
problem. Simply reoptimizing traditional solutions is not 
"good enough". 

B. DISTRIBUTED SYSTEMS 

A candidate solution is to apply the concept of distri- 
buted systems to the C3 problem in the tactical environment 
and to evaluate its performance against more traditionally 
structured systems. The actual use of distributed systems 
has not yet been implemented in an operational environment, 
but several new systems are beginning to display distributed 
features. The impetus for the move toward distributed 
systems came from the ARPANET experiment managed by the 
Defense Advanced Research Projects Agency (DARPA) . This net 
is an interconnection of computers and terminal users at 
widely distant locations which operate in a distributed 
manner using leased telephone lines as transmission links. 
This experiment has led to many new and valuable discoveries 
affecting both the public and private sectors of the tele- 
communications and data processing arenas. This type of 
technology advancement holds promise for tactical systems 
which plan to interconnect host computer systems with 
cabling. However, on a broader scale systems which use a 
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hybrid mix of cable, radio frequency (RF) links and other 
transmission systems are planned. Once again DARPA is in 
the lead in this area as they are presently conducting a 
field test of ARPANET technology with the Army at Fort Bragg, 
N.C. (XVIII Airborne Corps), and will very shortly begin a 
concurrent test of the impact on Command and Control struc- 
ture of a 'packet radio' data distribution system. (LAW 79) 

C. PLRS 

The Real Time Position Location and Reporting System 
(RTPLRS) is now under joint development by the Army and the 
Marine Corps and is to be the first operational tactical 
system with some "distributed" features. RTPLRS is a 
centrally managed network of ultra-high frequency (UHF) 
radios connected together by an AN/UYK-20 computer to provide 
ground forces with real time position location, navigation 
and targeting information. The central computer or Master 
Unit (MU) possesses global knowledge of the network routine 
assignments and station positions which it uses to answer 
queries from the network subscribers and assign them functions 
in support of its operation. The system uses a Time Division 
Multiple Access (TDMA) format to ensure orderly access to the 
system by all users. It employs a burst transmission (low 
duty cycle) mode of operation with a spread spectrum waveform 
combined with frequency hopping for electronic-counter-counter 
measures (ECCM) purposes. The distributed features evidenced 
by RTPLRS are a dual port adaptive routing technique which 
allows each node to have two paths to both the MU and its 
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next lower level of nodes. This dual port routing is 
managed by the MU using an algorithm which continuously 
evaluates each link assignment based upon present information 
for each link. The approach to link optimization is not a 
global one; rather, it is merely an ongoing attempt to guide 
the network toward a preferred state. Further each node is 
adaptively assigned cross-link (lateral) responsibilities to 
monitor other nodes within its radio line-of-sight (LOS) for 
time-of -arrival information to be used in range measurement 
calculations by the MU and for possible future dual port 
assignment changes. This automatic restructuring of the 
network in response to changes in connectivity and traffic 
flow patterns is a precursor of a network in which no MU is 
necessary and each node cooperates in a network management 
algorithm to allow distributed operations. (USA 77) Figure 
1 depicts the PLRS employment connectivity structure. 

D. ADDS 

As a next step in this area the Army is investigating 
the possibility of extending the RTPLRS technology into a 
data distribution system to be known as ADDS (Army Data 
Distribution System) . They have postulated that, by using 
a dual network approach or a multinet architecture, they can 
have many subnets of RTPLRS terminals operating under the 
control of a local net controller (NCU) with each of these 
under the control of a MU. They intend for such a system to 
serve a dual purpose. It will be available for standard 
position information and for a local data transfer network 
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as well. This technique begins to lessen the dependency 
upon the body of knowledge stored at the MU for each trans- 
action. The ADDS NUC's are envisioned to be flexible enough 
to allow operation of subnets without the MU should it become 
a casualty. (HUG 77) Figure 2 depicts a proposed ADDS 
employment connectivity, showing both the communication 
function as well as the position function. 

E. PLRS/JTIDS HYBRID 

An implementation of a similar but more capable concept 
is to be undertaken in the PLRS/JTID HYBRID (JTIDS: Joint 

Tactical Information Distribution System) . In this system 
the existing Enhanced PLRS User Unit (EPUU) will be utilized 
at lower echelon units (Battalion and below) to satisfy low 
data transfer requirements while the basic JTIDS terminal 
will be used at higher echelons (Brigade and Division) . 

(See Figure 3) . This will provide high volume/high speed- 
of-service with improved probability of delivery to the 
automated systems residing there. Combinational terminals 
will be placed at those units with requirements to pass data 
to both areas of the battlefield (Artillery Fire Detection 
Center) . As in ADDS NCU's will control PLRS subnets and a 
MU will oversee the PLRS/JTIDS interface and network problems 
The HYBRID is to provide "communications in support of battle 
field automated systems without supplanting the need for any 
of these systems or net radio". The HYBRID has been given an 
initial operational capability (IOC) target date of 1985. 

(USA 79) Figure 3 shows a proposed HYBRID employment concept 
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F. JTIDS 

The final step in the presently planned implementation 
of semi-distributed systems is with the deployment of JTIDS 
itself. This is planned to be stand-alone system to provide 
"an advanced communications, navigation and identification 
(CNI) system that will serve a wide variety of users. It is 
planned to use a low duty cycle, spread-spectrum waveform 
and advanced coding techniques to provide secure jam-resistant 
and low probability of exploitation (LPE) CNI functions. The 
system will implement a Multiple Access technique", choosing 
from between two competitors, "to provide various levels of 
connectivity (access) to user elements for simultaneous 
distribution and receipt of digital information." (ESL 78) 
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II. THE NATURE OF DISTRIBUTED SYSTEMS ARCHITECTURE 



A. BACKGROUND 

Architecture is the specification of the relationships 
between the parts of a system. A communications architec- 
ture for distributed systems includes the description of the 
formats, protocols, operational sequences, and logical 
structures for functions needed to achieve meaningful commu- 
nications. This should be true for units having wide ranging 
data input/output and processing capabilities which are 
members of the distributed network. Additionally these units 
may be physically separated by large distances and inter- 
connected by a variety of transmission media. (CYP 78) 

As technological advances permit us to have computer-like 
capability in almost every node in a network and the cost of 
logic and memory circuits diminishes, we are presented with 
increasing opportunities to assign a higher and higher degree 
of functional independence in physically separate nodes. 
Therefore the trend toward distributed networks interconnected 
by a variety of transmission media, e.g., wire or fiber optic 
cable, RF to satellites or LOS, etc., seems inevitable. But 
just because it appears that we can do it is not reason 
enough. What can the operational commander gain by distribu- 
tion and at what cost? Indeed what functions are themselves 
distributable and what trade-offs, if any, are there if we 
choose to centralize some while distributing others. Are 
certain of these distributable functions purely application 
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dependent? All these questions should be voiced early on in 
the architectural development of a distributed system designed 
for a particular application. We will now seek the answers 
to these and other pertinent questions that arise concerning 
issue — to distribute or not to distribute? 

B. DISTRIBUTED FUNCTION 

A function is said to be distributed if the same function 
can be executed at more than one node in a network, or if the 
function is not completely executable in a single node, and 
parts of that function are executed cooperatively in separate 
nodes. For each of these cases there are four key require- 
ments to ensure proper execution of the function. First, 
within a node all information must be stored which is needed 
by the distributed function. Second, the decision logic must 
be present within the node for that function. Third, the 
capability within a node to execute the logic of the distribu- 
ted function may be active or latent (be triggered externally). 
And finally, the capability to invoke the function from 
another node or to allocate work to that function from another 
node must exist. ( CYP 78) Figure 4 gives a framework for the 
concept of the nature of distribution. Note that the dimen- 
sions on the axes of the graph are hierarchies. Note also 
that by distributing our function over a geographic area this 
puts our system in the left vertical plane only. This is a 
good example of a partially distributed system. By including 
the concept of distributing the previous functions in time 
we can utilize the full space described by this graph. 
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C. TYPES OF DISTRIBUTED FUNCTIONS 



Potentially there are six types of functions located in 
a node of a distributed network. These are related to simi- 
lar functions in other nodes by a set of instructions known 
as internode protocols. The six classes of functions are: 

1. The management of application processing. 

2. The management of data that may be stored in a 
hierarchy of storage devices. 

3. The management of communications between nodes and 
of the dependent source/sink devices. 

4. User application program which utilize all of the 
first three management functions to some greater 
or lesser degree. 

5. An intermediate set of application subsystems. 

6. Input/output devices which may be dependent upon 
a node for their network access. 

It seems clear that ideally all nodes in a distributed 
network should contain the first three management functions 
in varying degrees, with the remaining ones resident where 
their use is dictated by the application. Thus the evolving 
data-processing network can be visualized as a chain of 
stations of various capabilities and at various locations 
moving in a random fashion linked together by diverse trans- 
mission means to perform a common function. Since each node 
will undoubtedly have some processing power (commonality of 
equipment may be a key issue here) , it should be the goal 
of the architecture to provide a common structure, increment- 
ally if required, to meet the varying needs for communication 
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among the users. (CYP 78) 



D. ADVANTAGES OF DISTRIBUTED FUNCTIONS 

In situations where more than one central site (command 
post) is required or where some of these sites may be 
augmented by specialized or supporting systems that may be 
remote from these sites, the advantages of distributed 
function in a network may apply. Clearly this seems to be 
the case here as supporting systems will be sprinkled through- 
out the battlefield. Specifically some possible gains might 
be : 

1. It may satisfy the need for local autonomy of function 
or flow control, local physical accessibility, local 
security or local application development. 

2 . It can provide greater availability to the users 

of the network by reducing dependence on relatively 
non-robust transmission links. 

3. It can provide better performance from the viewpoint 
of the users, as it shortens the response time for 
processes done locally where transmission delays or 
cycle access rates are an important consideration. 

4. It can reduce the data bandwidth required between 
nodes to reduce line costs and thus system costs. 

5. It utilizes the more efficient properties of distinct 
nodes when one is more suited to a particular appli- 
cation than another. 

6. It accommodates changes in local load patterns and 
applications without major disruption or expense to 
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7 . 



the system. 

It improves the quality of the data input leaving 
the source area with local aids such as edits, 
operator guidance and validity checks. 

8. It simplifies the programming support used for a 
given application by tailoring the support to that 
limited function. 

Obviously not all these advantages apply in all cases. 

In fact in some cases distributing functions poorly can have 
the opposite effect! For example, some processors will have 
superior response time only if their workload is low enough 
and they are not hamstrung by many inquiries to a central 
data base. Poor response time could result from a remote 
processor with slower speed than its host computer, depending 
upon their capacities and workloads. In some cases distribu- 
tion of function could increase system costs for input/output 
equipment and personnel. This would not be the case in the 
MCC application, however, as it is envisioned that the nodal 
terminal itself would serve as the in/out port for normal 
tactical operations manned by operations personnel. 

Thus the "optimum" distribution of function can be very 
application dependent as well as configuration dependent. 
Similarly the distribution of function should be adaptable, 
perhaps even adaptive, to meet changing user requirements. 
This diversity of trade-off makes it clear that many shades 
of distributed function could be considered as "optimum" for 
different situations. Thus an architecture for such a 
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distributed system must provide completely general communi- 
cability among network addressable units (nodes and the 

users they serve as well as external interfaces) . (CYP 78) 

In general, delays due to increased communications require- 
ments may also militate against a too widespread distribution 
of function. 
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III. HIERARCHICAL VERSUS DISTRIBUTED SYSTEMS 



A. HIERARCHICAL SYSTEM WEAKNESSES 

In order to be able to properly evaluate the strengths 
that distributed systems bring to the tactical C3 problem it 
is necessary to examine the weaknesses which hierarchical 
systems possess. 

Hierarchical networks and their supporting C3 systems are 
in general not richly connected . That is, because they 
emphasize the vertical order of senior/subordinate relation- 
ships they are unable to take advantage of possible lateral 
connectivities which might allow more overall system robust- 
ness. This is not to say that such lateral connectivities 
always exist or are never used in existing tactical C3 
systems rather; as a matter of course, the chain of communi- 
cations follows the chain of command. This principle has 
been valid for as long as there have been men under arms; 
however, that one's command relationships should be clearly 
delineated and fixed is not sufficient reason to also fix 
communications relationships in exactly the same way. 

Indeed fixing these patterns in such a way gives rise to the 
next general area of problem. 

The information mapping of such systems is a relatively 
simple task since the mapping function, in a mathematical 
sense, between the command structure and the communication 
links is one-to-one. Thus recovery of the communications 
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connectivity ensures recovery of the command relationships 
presently in force. Consider our system thus far. A few 
key nodes at each level of command, not richly connected 
vertically and most probably not laterally connected at all. 
Figure 5 shows such connectivity. Add to this picture the 
lack of mobility which present large C3 systems are bringing 
to the tactical environment and we simply fix the previous 
set of problems geographically for extended periods of time. 

The technological sword which has generated new and more 
capable automated systems is a double-edged one. While on 
the one hand it has allowed the garnering of great processing 
and display power for the commander, it has required that 
this power be grouped close to its power source in large 
containers (8x8x20 feet typically). These power sources and 
containers are sources of infrared (IR) emissions which can 
be used to our detriment by IR guided munitions or enenmy 
sensors. This doctrinal clustering of IR targets is an 
unacceptable tactical situation. 

Finally command post RF signatures and antenna "farms" 
have long been known to be undesirable side effects of our 
present system. New automated C3 systems do little to 
alleviate these problems. Remoting of antennas and emission 
control (EMCON) by communications and operations personnel 
have typically been our alternatives. There exists no 
doctrinal solution or support for the innovative commander 
who invents one to systematically overcome this problem on 
a broad scale. 
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B. DISTRIBUTED SYSTEM STRENGTHS 



How then do distributed systems allow us to plan to 
reduce the gravity of these C3 problems? From a connectivity 
standpoint distributed networks are designed to utilize every 
existing usable link within the network. Indeed while such 
a system does not itself imply any change in the command 
relationships it utilizes an adaptive process within the 
network structure to evaluate communications relationships 
such as link status and traffic flow patterns to adjust 
routing of messages according to a user specified set of 
"optimality" criteria. Thus the network becomes as richly 
connected as is possible in order to achieve maximum 
robustness . 

Certainly the continual or periodic generation of new 
routing solutions within the network in response to system 
requirements helps to erase the unity mapping function 
between command and communications structures. This feature 
with the ability to distribute functional capability through- 
out the network combine to allow us to build small and highly 
capable nodes (devices) which may be operated on the move in 
vehicles or manpack configurations. Such a system would 
utilize RF connectivities that changed as the nodes moved 
but for which the network management algorithm would compen- 
sate. This capability of mobile operations allows the 
commander a greater flexibility in the structuring of his 
command post(s) or groups beyond that presently achieved. 
Further it allows him a tactical flexibility seldom seen in 
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modern warfare. It seems implicit in distributed system 
design that he could disperse his staff over a wide physical 
area and still allow them to function effectively through 
the power of the network routing scheme and the use of power- 
ful display devices. While such a concept is radically 
different than those in use or planned to be used it serves 
to illustrate the fact that no longer will commanders' 
resourcefulness or ingenuity be limited by their C3 system. 

The battery or vehicular (jeep, truck, or tracked) power 
sources possible with such a system capable of mobile opera- 
tions have a secondary benefit in that they reduce the IR 
signature of such systems. Moreover, since there are many 
small nodes in the network it is more difficult to correlate 
their value with either their position or structural place- 
ment in the network. Figure 6 shows the connectivity of 
such a system. 

C. CONCLUSIONS 

It appears then that distributed systems have much to 
offer the commander in a tactical environment which he 
cannot get from a hierarchical system. However, we should 
not think that any technological breakthrough alone can 
completely solve our problems. It remains for the Commanders 
of tactical units to test and evaluate candidate distributed 
systems to ensure that they ultimately receive a system which 
truly meets their needs and not one which is just a replace- 
ment for the one presently in service. 
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FIGURE 6 . Possible Distributed Communications Linkage 
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IV. MOBILE COMMAND CONCEPT 



A. BACKGROUND 

As a result of a mission analysis conducted at the Naval 
Ocean Systems Center (NOSC) it was determined that a new 
post MTACCS concept for command and control capability within 
the Marine Corps was needed. The long leadtime involved in 
the Defense procurement cycle exacerbates the need for this 
concept to be formulated as early as possible. Conceptually 
the system(s) should be capable of mounted operation riding 
in LVTX's, jeeps, trucks and helicopters. Certain subsystems 
should be man transportable without major loss of function. 
That is, manpack components should be able to access the 
overall system with only a lower level of capability than 
the mobile systems, not with a complete absence of capability. 

Some of the initial design goals for these projected 
systems were that they should be totally reliable and not 
require field maintenance for a specified period of time, 
e.g., 30 to 60 days. Field maintenance in this context is 
anything other than a one for one replacement at the 
battalion level or lower. Certainly a module or "black box" 
concept at higher echelons is acceptable but the emphasis 
in this area is to relieve the tactical commander of the 
maintenance burden. The loss of any subsystem or component 
should not cause a catastrophic failure of the system. This 
concept of "soft fail" or graceful degradation in the face 
of battle damage is one which should be universally applied 
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to connectivity of these systems as well as to their 
functional capability. Finally the system(s) should be 
designed in such a way that the family of equipments used 
at battalion level form a building block of the regimental 
system and so on. (NAV 78) 

B . PROPOSED IMPLEMENTATION 

Doctrinally the Marine Corps is probably the most adapt- 
ive ground combat organization known today. The scope of the 
missions it is prepared to execute around the world is 
impressive. One reason for this wide range of capability is 
the way in which both its personnel and leaders are trained. 
"Use the 'book' when it applies and rewrite it when it 
doesn't.” This is a phrase often heard in many a Marine 
training exercise. This adaptability in the face of a new 
situation must be called upon again to handle this "new 
situation" . 

While present doctrine is flexible the command and control 
systems which exists to support it are not. Rather it is 
constrained by the nature and limited flexibility of present 
and planned systems. This was not by design but was an 
outgrowth of the previously addressed nature of hierarchical 
C3 architectures. Therefore, the following proposal for a 
distributed C3 architectural implementation is presented as 
a "straw man" to begin the inevitable dialogue necessary 
to achieve the realization of such technology in support of 
the post MTACCS C3 systems in the Marine Corps. 

The basic building block of a distributed network is a 
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nodal device which is capable of simultaneously housing 
communications (receive/ transmit) functions, computer 
(processing) functions and (in this application) visual 
display functions. In the manpack realization such a 
device would be capable of full network nodal status without 
the need for local storage of all applications programs 
resident within the network. Those routines required but 
not resident would be invoked from other nodes. For example, 
if a battalion level staff officer needed a report format to 
generate a daily report to his senior headquarters and as a 
result of operational requirements was using a smaller and 
less powerful device while he was enroute to a briefing, he 
could call up the required format from within the network by 
asking his staff section terminal at the headquarters to 
forward it to him. 

A battalion commander could utilize this capability to 
allow his staff to be subdivided, with himself, the executive 
officer and the operations officer as command group tactical 
officers (TACO's). Each group could be mounted and capable 
of assuming command for periods of time should a continuous 
combat situation arise or if the senior group were destroyed. 
In such a dispersal of staff functions the C.O., upon getting 
a new mission from his regimental headquarters, could send 
the situation map to the Operations Officer with a note on 
the new mission and a request for a plan. The S-3 would 
send back the map annotated with his plan and assignments. 

The C.O. would approve or modify it and send it off to the 
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Company C.O. 1 s. 

At a more senior level where more powerful nodal devices 
might be required the use of "add on" on-line storage could 
be accomplished in a variety of ways to allow more careful 
analysis of data and record keeping. This storage also 
allows devices access to a wide range of data without over- 
burdening the system with requests to recall it from other 
locations . 

As might be expected visual displays of liquid crystal 
or CRT type will play an important role in any such system. 
These will enable the commander to "see" large amounts of 
data meaningfully displayed. They should help all key 
command and staff personnel to actually reduce data transfer 
requirements to those necessary to doing the job at hand. 

(One picture if worth a thousand words) . Such action is 
vital to the success of any system but even more so in a 
distributed C3 system. Here the ability to transfer more 
data (i.e., more bandwidth) does not necessarily carry with 
it the need to do so. Needless overloading of the system 
with non-critical data merely slows or masks vital informa- 
tion. Although such a system is capable of both prioritizing 
messages and routing high priority traffic around low 
priority nodal blockages, these features should not be used 
to circumvent a real value inherent in automation. Reduction 
of traffic to the required messages by the use of good 
displays and selective use of preformatted messages is a 
desirable by-product of this implementation. As a first step 
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implementation. As a first step in this area the Army has 
recently published a Corps Information Flow (CIF) document 
which addresses this question of reducing overall data 
transfer requirements by identifying messages needed most in 
combat. These messages are broken down and formatted in 
matrices for ease of digital transmission. (CAC 79) 

In a distributed system context the operation of a command 
group at any level is essentially the same whether they are 
configured in a traditional command post arrangement or 
organized in teams of command and staff personnel mounted on 
a variety of platforms dispersed for protection against enemy 
attack. The power of the network and its ability to adapt to 
changes allows them the flexibility to operate just as effec- 
tively even though separated. Such tactical flexibility has 
rarely been placed in the hands of a commander. No longer 
need he be needlessly constrained by his C3 system to 
traditional solutions to emerging problems. His personal 
ingenuity will be rewarded by a system which can implement 
his creative solution almost immediately. 

More permanent installations such as Marine Air Wing C3 
facilities will be able to utilize more traditional linkage 
concepts such as Amphibious Assault Cable (26 twisted pair) 
or fiber optic cable in the future. The interface of these 
links with the RF structure of other networks will be a major 
design goal of such a distributed system to ensure maximum 
robustness throughout. This is presently an area of high 
technical risk and as such requires much integrated planning 
and study. 
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V. KEY TECHNICAL QUESTIONS 



A. NETWORK SYNCHRONIZATION 
1. Background 

To use distributed systems as a viable framework for 
a C3 architecture there are several technical considerations. 
Among these the first two which must be clearly demonstrated 
are those of network synchronization under any and all condi- 
tions and the ability to manage a volatile network in a 
distributed fashion. Without either of these capabilities 
distributed operation of a synchronous network is not possible. 

The synchronous operation which has been implied in 
the preceeding pages is an assumption not easily fulfilled in 
the distributed network context. While the question of 
synchronizations of various types (bit, word, carrier, etc.) 
have been rigorously addressed, each implies a building block 
approach to be extended throughout the network. A distributed 
system also needs such properties. The area in which work 
needs to be done for distributed systems is the area of over- 
all network synchronization. Is there a way to bring all the 
clocks in the network to agree on a common time? Is there an 
optimum approach for arbitrary networks which converges to a 
minimum number of transmissions necessary to accomplish this 
result? Can this be done without a time-out in network 
operations? 

One recent development in this area (FIN 79) presented 
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a protocol having the effect of synchronizing an entire 
network despite arbitrary finite delays. This protocol has 
several other features dealing with guaranteeing no lost or 
duplicate packets in such a network. These "protocols do 
require that the whole network be brought down (time out) 
for the duration of the resynch procedure." The author notes 
cogently that this "may be too high a price to pay for the 
failsafe properties in a practical network." In MCC it is 
too high a price. 

For a given network which is either completely or 
partially connected and operating standard synchronization 
procedures appear applicable. As a new member joins the 
network he aligns his clock to the network standard, or if 
he is an active member who senses his clock drifting he 
adjusts it upon receipt of a TODAD (Time of Day and Date) 
transmission. Note that implicit in both of these cases is 
the fact that there is an already accepted "absolute" time 
standard for the operating network. The source of such a 
standard in hierarchical structures like PLRS is the MU. In 
distributed networks there is no easy solution to this 
question. This is even more perplexing if the network is not 
operational and attempts to "cold start". Two cases appear 
viable for analysis. First: a distributed network with 

arbitrary connectivity operating synchronously. Second: a 

distributed network with arbitrary and temporarily unknown 
connectivity not operating. Such failure on a global network 
scale is envisioned as perhaps the result of a nuclear air 
burst over the Amphibious Operations Area. 
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2 . 



Two Cases 



For the first case it may reasonably be assumed 
that, in our tactical scenario, an operating system would 
have access to either a broadcast time standard (WWV) or 
for remote operations that nodes measure link propagation 
times and then pass time-of-day information from node to 
node. Thereafter, tracking of the carrier phase will allow 
all clocks to stay in synch. 

For the second case, however, very little may be 
assumed except that all clocks are set to a different time. 
The remaining network connectivities, for the failed net 
case, are unknown by all remaining operational nodes. Thus 
reconstruction of the network and synchronization seem to 
form a difficult pair of problems requiring a simultaneous 
solution. If a solution to them could be found it would 
also be viable in case one, which is clearly less difficult. 

3. Procedure for Network Synchronization 
a. General 

Let all nodes in the network be labeled alpha- 
betically with the procedure for doing so being arbitrary. 
Let the synchronization algorithm proceed in discrete time. 
That is, let each operation be (A t) seconds after the 
receipt of messages which will require the operation to 
take place. e.g., if (A t) were one millisecond this would 
remove all propagation factors (delays) of less than one 
hundred and eighty six (186) miles. Finally let each node 
have a counter which will begin counting each (A t) after 
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that node begins, or participates initially in, the pro- 
cedure. Thus a node is identified as 'Al' if it is node 
•A' and this is an initial transmission. Similarly if a 
relaying node is 'ft' and has not yet participated or 
initiated any TODAD ' s , upon receiving 'Al' it, 'R', would 
relay 'A2' signifying that it was in synch with Pseudo 
Master Clock (PMC) 'A' and this was a level '2' relay of 
that information. This counter is reset upon completion of 
the procedure or on receipt of an operational message. 

Such receipt would indicate operational net status, at least 
locally, and any further adjustments to clocks would be 
simply that, adjustments. 

As the nodes in a network attempt to reconstitute 
themselves they will begin to transmit in the broadcast mode 
to any other node who can hear them. Such transmissions will 
be TODAD information and will advise the recipient by their 
content that it is receiving a K'th level (iteration) trans- 
mission of such information. Whenever a node receives a 
TODAD identified as level k, it relabels it as a level k+1 
and retransmits it to all other neighbors. Any TODAD which 
might be received subsequently with a lower level identifica- 
tion is ignored. That is, the highest count wins when TODAD' s 
collide at nodes. This prevents nodes from entering a race 
track or tail chase condition. If two transmission collide 
with equal counter numbers then arbitrarily let the low 
alphabetical one be dominant. This prevents unsolvable 
dilemmas but does not inhibit the first node coming on the 
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air from gaining network synch the quickest. 

Thus in the simplest case the initial TODAD 
message proceeds outward through the network and once 
completed all clocks are in synch. A distributed network 
management algorithm could now begin to adaptively route 
messages through the network as in case one. Actually no 
notification of global acceptance of the pseudo master (PMC) 
is required as local (node to node) acceptance can begin the 
net management process and adjustments to PMC will not "undo” 
previously synchronized nodes nor are time-outs required to 
accomplish this. (see b. below) . 

Several other cases of interest are included in 
the PMC relay procedure. For example, what happens when two 
or more stations come up simultaneously and begin level one 
TODAD transmissions in adjacent or separate parts of the 
network? For the adjacent case the decisions needed are as 
previously stated: the subsequent level two relay will 

inform the competing nodes of the winner. At widely separa- 
ted distances within the network when two TODADS collide 
near the center with different levels of relay the highest 
level of relay is dominant having, by implication, already 
synchronized more nodes. The continual relay of the higher 
level TODAD will be heard by the losing node and promulgated 
as an adjustment down to his previously synched neighbors. 
Should two TODADS meet in the center of the network with 
equal levels of relay then rules are again applied with the 
subsequent changes propagating back as before. 
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b. Two Station Operation 



As a subset of the foregoing procedure the actual 



actions necessary for nodal timing were developed. If the 
receiver at each node is of a matched filter type and a 
spread spectrum waveform is employed then the following is 
applicable . 



the error £ between their clocks and to find the propagation 
time t between them. Let them do the following; where t' 
and t denote the time according to the clocks at A and B, 
respectively . 

GIVEN : 

Assume A is £ seconds behind B 
i . e . , when t' =0, t = £ > 0 , 
or t ' = t— £ 

A and B have identical cryptographic key-stream 
generators x( ) 

B has a programmable matched filter hg(t) 
with an impulse response set to: 



Consider two stations A and B needing to minimize 



V*) 




x(T-t) , 0 < t < NS 



0, elsewhere 



WHERE: 



T = time adjustment > £ + t + N<5 

(allows it to await event to come) 



N = Number of chips in sequence to be correlated 



6 = Chip duration in seconds 
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THEN: 






A transmits x(t') = 


x(t - e) 


B receives "f seconds 


later 


THEREFORE : 




r (t) = x( t - £ 


- x)*h(t) 


OO 

= _i x(a 


- £ - x) h(t - a) da 


OO 

= i x(a 

— OO 


- £ - t) x(T-t + a) da 


Which is maximum for 




-£ 


-x = T-t 


i.e. 


t = T + £ + X 



BUT: 



B knows T (offset introduced to await transmission) 



THEREFORE: 

B can calculate £ + x 



SIMILARLY: 



Assume A has a programmable matched filter: h^ ( t) 

with impulse response set to: 



h A (t) 



x ( T-t ' ) , 0< t'< N6 



0 , 



(Although his time origin is wrong, A's offset 
forward by T from the origin is not wrong.) 



NOW: 



Now in order to make it easy for A to determine , 
B transmits a signal advanced by £ + x 

A receives x seconds later 
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THUS : 



r ( t) = x ( t + e)*h(t) 

00 

~-L x ( a + £ ) h(t - a) da 
00 

= £ x(a + e) x(T-t + a) da 
which is maximum for e = T-t 
i.e. , t = T - e 

BUT: 

A knows T 
THERFORE: 

A knows e and can adjust his t' to be synchronized 

to t, to within an accuracy of 6 seconds. 

4 . Performance 

If only one PMC initiates a level one message then 
the number of levels required to synchronize the network is 
equal to the "diameter" of the connected graph formed by 
the existing connectivities within the network. Simply, 
the diameter of a graph is its maximum minimum (longest 
shortest) path between any two nodes in the network. This 
is sometimes known as the longest geodesic of a graph. 

(HAR 72) 

The amount of time required therefore is D x(d)t, 
where D is the network diameter and (£) t is the interval 
between successive level transmission. 
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B. DISTRIBUTED NETWORK MANAGEMENT 



1. Network Types 

Two basic types of networks are available for the 
transmission of digital data. The line or circuit-switched 
type and the message or packet-switched type. These are 
distinct techniques for communicating among the nodes of the 
network and combinations of these switching strategies are 
possible as in "pacuit" switching. (GER 79) 

Line-switched networks connect the source of the 
transmission and its destination (sink) by a communications 
path that is established at the beginning of the connection, 
and cancelled when the desired transfer of data is completed 
or when disrupted by a failure. In different routing 
strategies, described below, paths may remain fixed or be 
changed (not cancelled) during the existence of the connec- 
tion. Such a subset of line-switching is called "virtual" 
line-switching. Data is forwarded according to designated 
paths but messages corresponding to different connections 
are multiplexed together on each link so that the portion 
of the link capacity used by each message is varied 
according to its transmission requirements. 

Message-switched networks may utilize different 
paths from source to sink for almost every message generated. 
Such paths are not predetermined but are incrementally 
determined by intervening relay nodes. The selection of 
these neighbors is the heart of the routing strategy. 
Packet-switching is a fundamental subset of message switching 
in which messages are subdivided into a number of smaller 
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segments each of which is labelled by the network as a 
message destined for the same sink. (SID 79) 

2 . Control of Networks 

Two main approaches exist in the control of routing 
procedures within a network. There is the centralized 
approach, of which PLRS is a good example, and the decentral- 
ized or distributed approach of which presently ARPANET is 
an operating example. 

Little more needs to be said about centralized 
adaptive techniques beyond the observations noted in Chapter 
III. While achieving such goals as "optimal" routing is 
made easier, the centralized adaptive procedures are inher- 
ently weakened by the imposition of requirements for commu- 
nicating enormous amounts of status information back to the 
central node. In addition, the central memory (MU) could 
become separated from a portion of its subscriber network 
or vice versa due to link failures. Such a situation is 
highly undesirable in a tactical C3 system. Further the 
requirements for all routing to be centrally processed 
creates unbalanced demands on network link bandwidth which 
could limit the actual size of the supported network. The 
addition of sub-controllers (NCU) can help to alleviate 
some of the weakness that centralized systems display and 
the PLRS/JTIDS HYBRID uses such an approach. 

The distributed adaptive control schemes have none 
of the inherent inefficiency or unreliability of fixed 
routing nor the unreliability and size limits of centralized 
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control measures. Each node performs the necessary compu- 
tations to make routing decisions in conjunction with its 
adjacent nodes (neighbors) . This is usually manifested by 
a table of routing information being maintained at each node 
to identify the output links to be used from it for each 
destination in the network. Such tables could be updated 
periodically or asynchronously (as needed) by using the 
routing information each node collects internally and that 
which it receives from its neighbors. In an N node network 
each table can have up to N-l entries. Each is a measure of 
the estimated minimum distance from that node to every other 
node in the network as well as the neighbor to which a 
message is next relayed. 

Distributed routing systems are not all strengths 
with no weaknesses. Without any global knowledge of nodal 
status within the network loop freedom is difficult to 
guarantee. Failsafe operation of the network management 
procedure becomes much more difficult to achieve. These 
are some of the challenges of this fascinating subject. 

(SID 79) 

3. Routing Strategies 

Routing strategies can be measured by the yardstick 
of their dynamicism. Typically we have static, quasi-static 
and dynamic strategies. 

Purely static or deterministic routing sets up the 
rules which determine fractional traffic loads on each 
link prior to network operation. Further they determine 
actual path assignments from each node to each other node. 
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To do this they use knowledge about node location, connec- 
tivities, and link capacities as well as overall "loading" 
and requirements of the network. Fixed routing implies 
solutions to questions like how a message goes from A to B 
which are neither adaptive to changes in the network status 
or requirements, nor reliable enough for much practical use. 
Their simplicity, however, makes them appealing in the 
network design phase. 

On the other end of the spectrum are completely 
dynamic routing strategies which allow nearly continuous 
change of routes as a function of time and network loading 
conditions (traffic input, queue lengths, link and nodal 
failures and additions) . While responsiveness to network 
requirements is desirable the large amount of "overhead" 
per message through the network is an undesirable side 
effect. This overhead information is required in order to 
restructure addressing at each node in response to the 
changes in the network. 

Quasi-static routing is adaptive in nature but 
forbids continuous generation of new routing solutions. 

Here routing assignments may be modified only periodically 
or when a network need arises due to an extreme situation. 
Link and node failures or additions are typical events 
which would precipitate such action. In order to allow 
adaptivity the quasi-static routing procedure must be able 
to sense and react to changes in the network topology and 
loading conditions. Adaptivity to failures is very important 
in both our scenario and analytically in order to maintain 
a good grade of service to the network subscribers. (SID 79) 
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VI. A QUASI-STATIC DISTRIBUTED ROUTING PROTOCOL 



A. SYNOPSIS 

1. Background 

The second question to be investigated is the network 
management and routing of information in a distributed net- 
work. Without this capability synchronous operations 
previously guaranteed are of little advantage over present 
and planned systems. Implicit in the "distributed" opera- 
tions of our scenario are requirements that such operations 
include the properties of loop freedom for each destination 
at all times, adaptivity to changes in network flow and 
topology, and completely failsafe operation. Failsafe as 
used here means that given an arbitrary number and location 
of failures and/or additions in the network structure, the 
network recovers in finite time after the last change to 
provide routing paths between all remaining physically 
connected nodes. These properties are obtained using 
asynchronous computation of distributed status information 
in lieu of any one source having global topological know- 
ledge. The algorithm investigated here is due to Segall 
and Merlin. (SEG 79) 

2 . Procedures 

In the discussion that follows, the simplifying 
assumption is made that all traffic is destined to a single 
(sink) node. In practice, the procedure to be described 
would be iterated once for each possible sink in turn. 
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Each node knows only who its next node (preferred 
neighbor) on the route to a given destination is. Each 
node is responsible for choosing new preferred neighbors 
(Pi's) based upon updating its own routing tables. Such 
updates are coordinated by the protocol via control messages 
sent between adjacent nodes. It has been shown in (SEG 79) 
that for a given destination the set of routes maintained by 
the protocol are loop-free at all times, and that whenever 
no failures occur they form a spanning tree rooted at the 
sink (i.e., they connect all nodes). To each link in the 
network a strictly positive "distance" or weight is assigned 
which represents the cost of using that link. Such costs 
will vary with time according to link utilization and other 
factors such as queue length. The length of a path then is 
the sum of the distances of its links. 

Destinations (sinks) may asynchronously trigger the 
protocol to start new "update cycles" to generate new routing 
solutions based on new topology or loading conditions. Such 
adaptivity is a function of user specified goals, e.g., 
maximum throughput, minimum average delay and so forth. A 
cycle first propagates uptree while modifying the distance 
estimates from nodes to the destination and then comes back 
downtree while updating Pi's. Each cycle tends to find 
routes with short paths from each node to the destination. 
Assuming time-invariant link weights it finds the strict 
minimum within a finite number of iterations. 

When a link fails, notice of its failure is sent 
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down to the sink of the tree and up to disconnected nodes 
above it by the nodes adjacent to the failure. When such 
notice arrives at the sink it triggers a new update cycle 
and the protocol guarantees that within finite time all 
nodes physically connected, even if by presently unused 
links, to the sink will have a loop-free route to it. This 
property holds for multiple topological changes, and even 
if such changes occur while the protocol is active and an 
update is in progress. The recoverability of the protocol 
is achieved without employing a "time-out" in its operation. 
Such capability enhances not only the analysis and structured 
implementation of the protocol but gives it wide ranging 
applicability to tactical C3 problems. 

The protocol is intended for use in quasi-static 
routing of data in communications networks. Application of 
it to a line-switched network would imply that the routes 
generated would be used to assign paths to a new or disrupted 
call. The link weights might represent delays. Thus in 
steady state the minimum delay route for the new call is 
found. Should the links represent incremental delays then 
the path minimizes the average network delay. In message- 
switched systems where the PI is the first "hop" of the 
present best estimated route to the sink, increasing the 
fraction of messages sent over the shortest path seems 
entirely practical and the preferred routing method for our 
scenario. 
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One major benefit of this protocol is that it can 
replace the saturation or "flooding" used in some networks 
to locate mobile subscribers and to select the routing 
paths. This concept has the advantages of saturation routing 
without requiring time out while providing a route selected 
not only on the basis of instantaneous congestion but on 
averaged quantities. (SEG 79) 

3. Nodal Functions 

A detailed listing of the algorithm for an arbitrary 
node is provided in Appendix B. The actual number of dif- 
ferent operations performed by any node is small. A node 
receives and sends messages of four types. It updates its 
routing tables based on information contained in those 
messages . 

For example whenever a node i receives a message 
type MSG (see Table 1 Appendix B) from neighbor 1, the 
receiving node estimates and stores its distance through 1 
to the SINK. The required data to perform this updating 
is contained in the MSG message from 1. As another example 
when i has received MSG transmissions from all his neighbors 
i transmits an MSG to its PI and then determines a new PI 
if one exists based on the new information in its tables. 

This is done by choosing the new PI as the neighbor which 
provides minimum estimated distance from i to SINK. (SEG 79) 
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B. SIMULATION 



1. Method 

"A theory has only the alternative of being right 
or wrong. A model has a third possibility; it 
may be right but irrelevant. " 

<v 

Manfred Eigen 

The simulation conducted of the protocol discussed 
above took the form of an examination of the incremental 
steps the algorithm performed in a volatile operating 
network. Each facet (capability) of the protocol was 
demonstrated, not by way of proving any separate character- 
istic, but rather to examine the data storage requirements 
necessary to support execution of the algorithm at any node 
in a network. Rigorous analytical proofs of all stated 
properties of the algorithm are included in (SID 79) . In 
order to display the dynamic capability of the protocol in 
a static display, like a thesis, a method of data presenta- 
tion was developed to examine the data status and message 
processing and handling capabilities of the algorithm. A 
computer simulation of it was developed in SIMSCRIPT II. 5 
(Appendix A) . This was intended to be a completely general 
representation of the statement of the functioning algorithm 
whose operation was summarized by the authors as a finite 
state machine (FSM) in which transitions between states 
were triggered by the arrival of control messages from 
neighbors and actions taken as a result of those transitions. 

2 . Results 

In answer to the question of data storage require- 
ments in support of the distributed network management 
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algorithm, both analytical techniques and simulation 
results were used. By examining the data stored by the 
algorithm at each node and noting the nature of each data 
set it was quickly determined that of the ten data elements 
six varied linearly with the number of nodes (N) in the 
network. Four were found to be directly proportional to 
both the number of nodes and the number of links (L) 
connecting them. Thus ((6N) + (4NL)) data elements need to 
be maintained by each node in support of the algorithm 
operation. This also proves to be the number of data 
elements stored by each node in the simulation. See Table 
1 of Appendix C for a list of the variables of the algorithm. 
While each data element is in general less than five char- 
acters in length the total characters stored in a node may 
become excessive for a microprocessor application. This 
would be the case in a very large network, more than fifty 
nodes for example. In such a network with one hundred links 
the nodal storage requirements for protocol related data 
alone is over 11,000 characters. Superimposed upon this 
would be application programs and other local protocols 
necessary for network operations and use by subscriber as 
well as message buffers for communications functions plus 
ROM coding of protocol instructions. 

These results begin to indicate that some sub-network 
structure might be beneficial to the storage requirements of 
individual nodes as well as to network responsiveness. By 
limiting the bulk memory requirements their size can be 
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kept to a minimum for ease of transportation in a highly 
mobile environment. They must still be capable of full 
support of their subscribers through the use of distributed 
functions. So a trade off may need to be made in their size 
and weight versus the desire for completely distributed 
operations . 

The ability of the algorithm to function in a 
dynamic network was not doubted at the outset. The proofs 
which are appended to the source document were clear and 
thorough leaving little doubt in the readers mind that such 
a system could be implemented. The purpose of the simula- 
tion therefore was not by way of a proof but rather a 
demonstration. It is intended for operational and design 
personnel to make them aware that such a powerful algorithm 
exists and can possibly be tailored to the needs of tactical 
C3 systems of the future. The various capabilities of the 
algorithm to handle network failures and additions in 
conjunction with routine network management functions were 
demonstrated. 

The data which follows was generated from the 
simulation listing of Appendix A. The initial status of 
all links shown were operational and the weights are as 
indicated in the network diagrams. The changes in link 
weights are specified in the narrative explanation of the 
data on each page. The final tree after simulation is also 
provided for information. 
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The fact that all simulation runs converged to short 
path solutions in one iteration should not be misconstrued 
to mean that all such changes in network status can be 
solved in one pass. Rather in these simple examples the 
singularity of failures/changes without compunding ones 
allowed the algorithm to operate in its most efficient 
manner without need for second or subsequent trys to find 
the optimum solution as would probably be necessary in more 
complex and volatile networks. 
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N = 7 
L = 11 



NETWORK 



1 



1. Number of transmissions required to complete 



update cycle (root to root) 22 

2. Number of iterations/ transmissions to find 
a new short path solution given: 

A. Link failure in (4,5) 1/24 

B. Link weight change in (4,5) from 1 to 4 1/24 

3. Number of iterations/transmissions to find 
a new short path solution given: 

A. Link failure near root (2,4) 1/28 

B. Link failure in a branch (6,7) 1/24 
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N = 11 
L = 17 



NETWORK - 2 



1. Number of transmissions required to complete 
an update cycle (root to root) 

2. Number of iterations/transmissions to find 
a new short path solution given: 

A. Link failure in (4,7) 

B. Link weight change in (5,8) from 3 to 6 

3. Number of iterations/transmissions to find 
a new short path solution given: 

A. Link failure near root (1,4) 

B. Link failure in branch (7,11) 



34 



1/43 

1/36 



1/52 

1/34 
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N = 14 
L = 24 



NETWORK - 3 



1. Number of transmissions required to complete 



an update cycle (root to root) 48 

2. Number of iterations/transmissions to find 
a new short path solution given: 

A. Link failure in (3,8) 1/62 

B. Link weight change in (4,5) from 2 to 6 1/50 

3. Number of iterations/transmissions to find 
a new short path solution given: 

A. Link failure near root (1,3) 1/66 

B. Link failure in branch (11, 14) 1/48 
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VII. CONCLUSIONS AND RECOMMENDATIONS 



A. CONCLUSIONS 

The development of a distributed C3 system as a post 
MTACCS concept appears both desirable and possible. In the 
face of the Radio Electronic Combat (REC) threat few detrac- 
tors can say that such system capabilities are not desirable. 
No commander would say that he did not want such flexible 
support for his tactical decisions in a mobile and destruc- 
tive conflict. 

It is possible because it has been demonstrated that the 
two greatest initial problems which blocked distributed 
system implementation in a tactical environment have been 
removed. The key concept is the ability to manage the 
network in a loop- free and failsafe manner. Secondary to 
this is the ability to guarantee network synchronization 
given temporary catastrophic failure in synchronous network 
operations. Hierarchical C3 systems may be an architecture 
whose time is passing, and it remains only for operational 
and design personnel to sufficiently delineate the desired 
capabilities of a system like MCC to begin its actual 
deve lopment . 

The problem of desired sub-network structure within a 
distributed system appears to be a series of trade-offs 
between the amount of structure versus the amount of distri- 
bution. Each facet brings with it those strengths and 



61 



weaknesses previously discussed. It is desired to retain as 
many of the advantages of distributed operation as possible 
while eliminating the unmanageably large delay times 
required for new solutions to be generated in large networks. 
Further it is desirable not to build in so much structure 
that the problems associated with present architectures 
resurface. Whether such trade-offs would be a function of 
the location of nodes, connectivity, or other factors either 
singly or in combination is unknown. A recent paper on 
shortest path algorithms in communications (YEN 79) networks 
has suggested a framework for a possible solution to this 
problem. Its description follows, but like the MCC concept 
presented previously, this proposal is a subject for further 
investigation . 

This algorithm operates under the following assumptions. 
Each pair of nodes is connected by at least one pair of . 
bi-directional links. The delays on these links are dif- 
ferent for different directions of travel (queue lengths 
are assumed to constitute the major delay) . All timing and 
service messages are jumped to the head of the message queue 
to ensure proper timing on outbound links. 

At time T(0) the sink transmits to its neighbors who 
hear and record the message. They in turn transmit to their 
neighbors at t(0) + £ (X) , where £(X) = local queue delay 

time for the link to the sink. Each of their neighbors 
upon hearing this transmission, waits until T(0) + & (X) + 
£(Y), where £ (Y) = their local queue delay toward the sink, 
then they transmit as before. (See Figure 7.) 
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FIGURE 7. A Shortest Path Algorithm 
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Should a node be the recipient of two or more trans- 
missions from its neighbors it will of course forward only 
the earliest received as this will ensure those above it in 
the network of the shortest path to the sink. 

Since each node waits until time t(0) + (all cumulative 
6's) before transmitting, it is assured of never receiving 
a later message with a shorter path to the sink. Further, 
each node will only be required to transmit once, and no 
acknowledgement is required. (YEN 79) 

The proposal for sub-network structure is a direct 
result of the previously discussed work (See also Figure 8 
which follows) . Let the set of all nodes be known as the 
set (Z) . Let this be divided into regions (R(i)) and let 
these be further divided into areas (A(j)) . Let each area 
be divided into sectors (s(k)), and let each of these be 
further divided into sub-sectors (Q(l)). Let the divisor 
be four. With four nodes in each Q(l) the network capacity 
is 1024 nodes. Thus this example translates to a notional 
two Marine Amphibious Brigade (MAB) size network. (This 
example assumes a PLRS notional MAB of 400 nodes) . 

Since each station only transmits once to achieve local 
synchronization let those stations (nodes) in R(1).A(3). 
S(2).Q(4) (See Figure 8) do this to achieve this state. 

Let one of these nodes be designated, by an as yet undeter- 
mined method, as the Primary Station (PS) . Let there be one 
PS for each level of division, e.g., there is one PS acting 
for R(l) in A(l) and there is one PS acting for A(3) in S(4) 



64 




(Z) = set of all nodes 
R(l) = Region 
A ( j ) = Area 
S(k) = Sector 
Q ( 1) = Sub-sector 

A = Primary station 



FIGURE 8. Distributed Network Sub-structure 
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and for S(2) in Q(4). Thus there is a PS acting for each 
level of division but resident one level below that level. 
Let each station in each level quadrant find shortest path 
solutions to each PS in the next level higher quadrant, Q 
to S for example. Then let each station find the shortest 
path to each PS two levels above itself, Q to A for example. 
Finally let each node find the shortest path to each PS 3 
levels above itself, Q to R. The algorithm indicates that 
this procedure take four times the logarithm (base 4) of N 
transmissions by each node. 

Then as indicated in Figure 8 when an arbitrary node X 
wanted to send a message to R( 1) . A ( 3) . S (2 ) . Q ( 4) it would 
proceed by finding the shortest path out of its Region. 

Then the message would be routed in R(4) towards R(l) via 
the shortest path to the PS in A(l) of R(l) . Finally upon 
crossing into R(l) the local nodes there would have 
knowledge of shortest paths to A (3) and so divert the 
message from the PS in A(l) toward A(3) and upon crossing 
that boundary similarly to S(2) and finally to Q(4). 

While it is recognized that this algorithm and its 
presentation in this example does not represent the 
strictest short path solution its virtues are simplicity 
and low number of transmission as well as small storage 
requirements for routing information required to implement 
it while still achieving a short path in such a network. 
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B. RECOMMENDATIONS 



It is recommended that the Marine Corps closely monitor 
the present test of ARPANET technology at Fort Bragg and 
maintain close liaison with the Army in the development and 
implementation of the ADDS packet-radio test to follow. 

It is further recommended that the Marine Corps begin to 
examine in greater detail the concept of sub-network 
structure in a distributed C3 system with a view towards 
optimizing the trade-off in node size and capability. As 
a possible source for such investigation it is also 
recommended that the officer-graduate students of the 
Marine Corps be utilized as a resource to accomplish this 
and other related tasks in the future development of the 
post MTACCS C3 systems. This group of individuals possess 
the time and resources to provide valuable input to such 
decisions and their efficient use by the Marine Corps could 
help to limit the initial cost of the development of such 
systems in the future. 
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APPENDIX A 

DISTRIBUTED NETWORK MANAGEMENT SIMULATION LISTING 



// EXEC SIM25CLG 
//SIM.SYSIN DC * 

"THIS IS THE E 4 S IC SIMULATION OF MY THESIS'* 

"IT IS C5SIGNEC TC SIMULATE THE DISTRIBUTED NETWORK " 
"MANAGEMENT ALGORITHM O c ADRIAN SSGALL AT ITS NCST eASIC" 
"LEVEL FOP ONE TREE (EVENTUALLY FOR MULTIPLE TREES)" 

"THE ENTIRE SIMULATION WILL SHCW THE CAPABILITY OF THE" 
"ALGORITHM TO HANDLE LINK FAILURES , NETWORK ADDITIONS " 
"AND ROUTINE NETWORK MANAGEMENT. IT IS DESIRED TO BE ABLE" 
"TO DETERMINE HCW MUCH STORAGE OF NETWORK MANAGEMENT DATA" 
"IS RE CURIE C EY EACH NODE/ AND HOW POWERFUL A PROCESSOR" 
"THIS ALGORITHM MIGHT RECUIPE IN A MAN-PACK REALIZATION." 
"OThER CONSIDERATIONS ARE WHAT IMPACT SUCH A CAPABILITY, " 
" 1.5. CISTRIEUTEC NETWORK MANAGEMENT; EXTR APOLATEC " 

"TO DISTRIBUTED C C MMUNIC AT I CNS SYSTEMS , MI GHT HAVE ON THE " 
"COMMANC AND CONTROL STRUCTURE OF THE MARINS CORPS IN THE" 
"1990'S WITH SPECIFIC ATTENTION TO THE COMMUNICATIONS " 

• 'NETWORK IMPACT. ' ' 



PREAMBLE 

NORMALLY MCCE IS INTEGER 
GENERATE LIST FCLTINSS 
PERMANENT ENTITIES 

EVERY NODE HAS A PI, A DI, A MAXI, AN NI,A CT, A NBR , 

A STATE, AND CWNS = CLS 
EVERY SINK HAS A SMAXI,A SCT, 

A SSTATE, AND CWNS THE CUEUS 
TEMPORARY ENTITIES 

EVERY MESSAGE HAS A SNK , AN L , AN M , A D, AN I, A TYPE, 

AN ORIGIN, A OEST AND *AY BELONG TO THE QUE ANC 
THE QUEUE 

DEFINE FIL ANC SFIL AS 2-D I MENS I CNAL , ALPHA ARRAYS 
OEFINE N ,K , MM AS INTEGER VARIABLES 

DEFINE ZIL, CILL, OIL, NIL* SOIL, SNIL AS 2-DIMSNSI ONAL , INTEGER 
ARRAYS 

DEFINE STATE , SSTATE , AMD TYPE AS ALPHA VARIABLES 

DEFINE CONNECTIVITY AND TREE AS 2-DIMENSIONA L, INTEGER ARRAYS 

EXTERNAL EVENTS APE START . U F ,NODE . ADD, L INK . FAI L 

EVENT NOTICES I NCLLD C STOP . S I MUL ATI ON , OUT. PUT 

EVERY RECEIVE. A. MESSAGE HAS A TEXT AND A HOME 

END 

MAIM 

READ N "THE DIMENSIONS OF THE CONNECTIVITY ARRAY" 

RESERVE CCNNEC7IVI7Y<*,*) AS N BY N 

READ CONNECTIVITY "READS 17 A ROW AT A TIME FROM CARDS" 
RESERVE CIL, ZIL , CILL, NIL, SOIL, AND SNIL AS N BY N 
READ CIL 

RESERVE FIL ANC SFIL AS N BY N 
READ FIL "UP/CCWN LINK STATUS'' 

READ MM "THE CIMENSICNS OF THE CIRECT5D TREE, 1 TO START" 
RESERVE TREE AS MM BY MM 

READ TREE "READS IT A ROW AT A TIME FROM CARDS" 

READ K "SIZE CF NETWORK' • 

CREATE EVERY SINK(K) "SIZES NODE ATTRIBUTES" 

CREATE EVERY NCCEIK) "-DO-" 

FOR EACH NODE LET STA T E ( NODE ) = " S 1" 

FOR EVERY NCOS FEAC P I (NOD E ) , C I ( NODE ) 

FOR EVERY NCOS PEAC NAXI(NOCE) 

SCHEDULE A CUT. PIT AT TIMF.V 4 39.9 

SCHEDULE A STC P . S I MUL ATI ON AT TIME.V 4 - 40.0 "LONGEST 4 -" 
START SIMULATION ' 'GOES TO FIRST EVENT, CARD AFTER CATA •• 
END 

EVENT START. LF 

PRINT 1 LINE WITH TIME.V AS FALLOWS 

EVENT START. LF ENTERED, THE TIME IS ****** 

READ R VCR 
CREATE A MESSAGE 

LET TYF5 (MESSAGE )="REQ" 

READ SNK (MESSAGE) , DEST ( MESS A GE ) ,C( MESSAGE) ,M ( MESSAGE ), 
CFIGIN( MESSAGE) 
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SCHEDULE A RECEIVE. A. MESSAGE GIVEN MESSAGE AND PVCR AT 
TINE.V -*• l.C 

LIST ATTRIBUTES CF MESSAGE 

RETURN 

END 

EVENT NGCE.ADC 

PRINT 1 LINE WITH TINE.V AS FCLLCWS 

EVENT NCOS. ACC ENTERED, THE TIME IS ***.** 

READ F VCR 
CREATE A MESSAGE 

LET TYP£(MESSAGE) = ,, WAKE ,, 

READ SM<( MESSAGE) , CEST ( MES S A C-E ), C ( MESSAGE ), M ( MESS AG E ) , 
ORIGIN(M5SSAGE ) 

SCHEDULE A REC5 IVE .A. MESSAGE GIVEN MESSAGE AND DEST AT 
TINE.V + 2.2 
RETURN 
END 

EVENT LINK. FAIL 

PRINT 1 LINE WITH TINE.V AS FCLLCWS 

EVENT LINK. FAIL ENTERED, THE TIME IS ***.** 

READ F VC R 

CREATE A MESSAGE • 'MESSAGE CCWN'-TREE TO SINK" 

LET TYPE(MESSAC-E)="FAIL" 

READ SNK (MESSAGE) , C EST ( MES S A GE ) , C ( MESSAGE ), M ( MESS AC- E ) , 

ORI GIN( MESSAGE ) 

SCHEDULE A RECE IVE .A. MESSAGE GIVEN MESSAGE AND DEST AT 
TINE.V + 2.3 
READ R VCR 

CREATE A MESSAGE "MESSAGE U°-TREE TO DISCONNECTED NODES" 
LET TYPE ( MESSAGE) ="FAI L” 

READ SNK (MESSAGE ), DEST ( ME S SAGE ) ,C( MESSAGE) ,M< MESSAGE I, 
ORIGIN(MESSAGE) 

SCHEDULE A RECE I VS . A . MESSAG : GIVEN MESSAGE AND CEST AT 
TINE.V + 2.31 
RETURN 
END 

EVENT RECEIVE. A .MESSAGE GIVEN COPY AND CATCHER 
PRINT 1 LINE WITH TINE.V AS e OLLCWS 

EVENT RECEIVE. A. MESSAGE ENTERED, THE TIME IS ***.** 

• BAS I C * 

LET NCCE=CATCFER 

IF TYPE(COPY) = " M S G 11 AMO STATP(NCDE) =«'S1 •’ , 

LET NIL (DEST (COPY) , OR I G IN ( COPY ) )=M(COPY) 

LET C I LL ( NCDE ,CR I GI N ( COPY ) ) =C ( COPY ) +DI L ( NO CE , CR I G IN ( COPY ) ) 
IF FILtNCDE .ORIGIN (COPY) )=" READY" 

LET FIL(NCCE, ORIGIN (COPY) )="UP" 

ALWAYS 

IF CRIGINICCPV ) = PI (NODE) , 

LET CTtNCCE ) = 0 
LET STATE (NGCS)- , 'S2 M 
GO TO BFSM12 
ELSE 
RETURN 



ELSE 

IF TY PE (COPY ) = "MSG M AND DE ST (COPY ) =SNK ( COP Y ) AND 

SFIL(NCDE,CRIGIN (COPY) ) = ”HRrt« AND 

STATE ( NODE ) = ” S 2 

LET ST ATE ( NODE ) = "S 1" 

PRINT 1 LINE AS FCLLCWS 
FFOPEP COMPLETION 
RETURN 
ELSE 

IF TY PE ( COPY ) = M M S G " AND ST ATE (NODE ) ="S2", 

LET NIL (DEST (COPY) , ORIGIN (COPY) )=M(COPY) 

LET DILLtNCDc ,CRIGIN(CCFY) )=D (COPY) + 

D I L (NODE , (CP IG IN (COPY) ) ) 

FOR Ii=i TC N , WITH CONNECTIVITY (NODE, 1 1 ) GT 0 
ANC NIL (NODE ,1 I ) LE 0, 

FINC TFE FIRST CASE 
IF NCNE , GC TC EFSM21 
ELSE 
RETURN 
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ELSE 

IF TY P E ( CO PY ) = " F A I L" AND STATS (NODE ) = "S1 '• AMD 
GRIGIMCGFY )-FI (NCOS) 

LET FIL(MCDE,CRIGIN(COPY) ) = "CCWN" 

LET STATE (NCCE ) = "S3" 

LET CT ( NODE ) =0 
GO 10 BFSM12 
ELSE 

IF TYPE(COPY)="FAIL" AND STATE (NCOS )="S2" AND 
ORIGIN (COPY ) = P I ( NODS ) 

LET FIL (NCCE ,CR 3 GIN (COPY ) )="DCWN" 

LET STATE (NCCE) ="S3" 

LET CT ( NODE )=C 
GC 1G BFSN22 
ELSE 

IF TYPS( COPY)* "WAKE" AND PI(NODE)=0 AND DI ( NODS )=99999 
FCF KS= 1 TC N » WITH P IL ( NCCE » KS ) ="UP" AND 
NIL(NCD5,KS) = MAM(NCDS) 

CCM PUTS MINCIL AS THE MINIMUM OF DTL(NODE,KS) 

LET PI (NCDE ) = K S 

LET NI<NGOS)=NAXl(NGDS) 

LET Cl (NODE )=MINCIL 

GC 1C 3FSM32 "REATTACHING NCCSS ONLY 1 1 
ELSE 

IF TYPE (COPY )="WAKS" AMO PI(NCOE) NE 0 "UPDATE TABLES" 
LET CCNNr CT I V 1 1 V (NCOS , CR I G IN ( MESSAGE ) ) = D( MESSAGE I 
LET CCNNSCTIVITY (ORIGIN (MESSAGE ) , NODE ) =D ( ME S SAGS ) 
RETURN 
ELSE 

IF TYPEtCOPY )=«RSC" AND DEST(COPY) = SNK(COPY) 

FOR K Z = 1 TO N, WITH CONNECTIVITY (NOOS»KZ) GT 0 
DO 

CREATE A MESSAGE "START NSW UPDATE CYCLE" 

LET TYPE (MESSAGE) = "MSG" 

LET M(MSSSAGE) = MAXI (NCCE) + 1 
LET SNK (MESSAGE) 55 SNK(CGPY) 

LET ORIGIN (MESSAGE) =DE$*MCOPY ) 

LET DEST ( M EE SAGE ) =KZ 

LET SFIL IN OCEt ORIGIN (CCPY) ) = "HRD" 

SCH5DLLE A RECEIVE. A. MESSAGE GIVEN MESSAGE AND CSST AT 
TIME.V 4 l.C 

LIST ATTRIBUTES CF MESSAGE 
LET MAM (NCDE) = M (MESSAGE) 

LOOP 

DESTRCY MESSAGE CALLED COPY 
LET S7AT£(NODE)="S2" 

RETURN 

ELSE 

IF TYPEtCOPY) =' , REC" AND PI (NCCE) NE 0 
CREATE A MESSAGE 

LET TYPE ( M ESS AGE ) = "REC" 

LET SNK(MESSAG5)=SNK(CCPY) 

LET ORIGIN (MES$AGE)=DEST(COPY) 

LET M(M5SSAGc)=N(C0PY) 

LET DEST(MESSAGS)=PI (DEST(CCPY) ) 

SCH5DLLE A R5C S I VE . A . MESS AG E GIVEN MESSAGE AND DEST AT 
TIME.V + l.C 

LIST ATTRIBUTES CF MESSAGE 
DESTRCT MESSAGE CALLED COPY 
RETURN 
ELSE 

FRINT 1 LINS AS FCLLCWS 

ERROR INCORRECT MESSAGE TYPE RECEIVED 

RETURN 

•BFSM12' 

PRINT 1 LINE WITH TIME.V AS FCLLCWS 

BFSM12 ENTEPECt THE TIME IS ***.** 

FOR IJ= 1 TC N* 

WITH NIL (NODE » I J ) EC M(COPY) AND FI L ( NCDE , I J ) = "UP" AND 
CILL(NCDE, IJ > C-T 0 

COMPLTE MINDIL AS THE MINIMUM CF D I LL ( NODE, I J ) 

PRINT 1 LINE WITH MINDIL AS FOLLOWS 
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MINDIL = *** 

IF M(CCPY) GT MAXI(NODS) , 

LET MAXI(NCCE)=M(CCPY) 

ALWAYS 

FOR KK = 1 TO N , 

WITH CCNNEC71VITY(NOOS,KK ) GT 0 AND 
KK NS PI ( NCCE ) AND FIL(NCCE»KK)="UP" 

DO 

CREATE A MESSAGE 

LET DEST ( ME S SAGE ) = KK 
LET TYPE ( ME SS AG E ) = "MSG" 

LET SNK (MESSAGE )= SNK(CCFY) 

LET D ( MES SAGE ) = Cl (NODE) 

LET M(M5SSAGE)=NAX!(NODE) 

LET CRIC-IN (MESSAGED DSST(COPY) 

LIST ATTRIBUTES CF MESSAGE 

SCHEDULE A R S C E I VS . A. MESSAGE GIVEN MESSAGE AND DEST AT 
TIME.V -» l.C 
LOOP 

FCR IX = 1 TO N, WITH CONNECTIVITY (NODE, IX) GT 0 
ANC NIL (NCCE , IX ) LE 0, FINC THE FIRST CASE 
IF NONE, C-G TC EFSM21 
ALWAYS 

DESTRCY MESSAGE CALLEC COPY 
RETURN 
' BFSM21 ' 

PRINT 3 LINE WITH TIME.V AS FCLLCWS 
BFSM21 ENTEREC, THE TIME IS ***.** 

FCR J K =1 TC N , VITF D I LL ( NCC E , JK ) GT 0 

COMPUTE NEWPI AS THE MINIMUM (JK) CF DI LL ( NODE , JK ) 

LET PI (NCCE ) = NEWFI 

FOR K J= 1 TC N, WITH F I L ( NOCE , K J ) ="U P" 

LET NIK NOCE » K J ) =C 

LET C T (NCDE ) = 1 

LET STATE(N0CE)="S1" 

CREATE A MESSAGE 

LET TYFE(M£SSACE)="MSG» 

LET SNK( MESSAGE )=$NK(COPY) 

LET C (MESSAGE ) = C I (NOCE) 

LET CRIGIN(NE$SAGE)=OEST(CCFY) 

LET M(MESSAGE )=MAXI(NCDE) 

LET CEST(MESSAGE ) =FI (CATCHER ) 

SCHEDULE A P ECE I VS . A. MESSAGE GIVEN MESSAGE ANO CEST AT 
TIME.V -» l.C 

LIST ATTRIBUTES CF MESSAGE 
CESTRO MElSAC-E CALLED COPY 
RETURN 
• BFS M IS * 

PRINT 3 LINE WITH TIME.V AS FCLLCWS 

BFSM12 ENTEREC THE TIME IS ***.** 

LET D I ( NCDE ) =SS999 ''APFRCX INFINITY'* 

LET NIL (NCDE .GRIG IN (COPY) ) = M(COPY) 

FCR IP = 1 TC N, WITH F IL ( NO C E , I P ) =" READY" 

IF ML(NCDE,IP) GT ZIL(NCCE,IF) 

LET FIL (NOCE , IP )="UP" 

LET NIL (NCCE ,IF )=0 
ALWAYS 

FOR IC=1 TC N, WITH CONNECT I VITY (NODE , I Q ) GT 0 
ANC IQ NE P I (NCDE ) AND F I L ( NC CE , I C ) ="UP" 

CO 

CREATE A MESSAGE 

LET D(MESSAC-E) = CI (NODE) 

LET M(MESSAGE)=MAXI (NODE) 

LET CRIGIN (MESSAGE) 3 NOCE 
LET DEST (MESSAGE)- IQ 
LET SNK (MESSAGE ) = SNK(COPY ) 

SCHEDULE A R EC E I V E . A. MESS A GS GIVEN MESSAGE AND IQ AT 
TIME.V -♦ l.C 
LOOP 

DESTRCY MESSAGE CALLEC COPY 
LET Pi (NODE ) =C 
LET CT (NCDE ) = 1 
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RETURN 
' BFS N 22 « 

PRINT 1 LINE WITH TINE.V AS FCLLCWS 

EFSM22 ENTERED THE TINE IS ***.** 

LET C I (NODE ) = 99S99 ”APROX INFINITY” 

LET ML (NCOS ,CPIGIN< COPY) ) = N (COPY) 

FOR IP = 1 TO N, VsITh FlL(NODE,IR)="READY" 

IF ML(NCDE,IR) GT ZI L ( NCDE , IR ) 

LET F IL ( NCDE ,IRJ="UP" 

LET ML (NCDE, IR ) = C 
ALWAYS 

FOR KC = 1 TO N, WITH CQNNECTI VI TY (NODE , KQ) GT 0 
AND KC NE PI(NCCE) AND F I L( NIOCE , KQ ) = "UP " 

DO 

CREATE A MESSAGE 

LET C (MESSAGE I = C I (NODE) 

LET M(M5SSAGEi=NAXI(N0DE) 

LET CRIGIA (MESSAGE )= NOC5 
LET DEST (MESSAGE )=KQ 
LET SNK (MESSAGE ) =$NK(COPY ) 

SCHECULE A RECE IVE .A. MESS AGE GIVEN MESSAGE AND DEST AT 
TINE.V + l.C 
LOOP 

CESTRGV MESSAGE CALLED COPY 

LET P I ( NCDE ) =0 

LET CT (NODE ) = 1 

RETURN 

•BFSM22* 

PRINT 1 LINE WITH TINE.V AS FOLLOWS 

EFSN32 ENTEFEC, THE TINE IS ***.** 

FOR K R = 1 TO N. WITH CONNECTIVITY (NODE.KR I GT 0 
ANC KR NE PI(NOCE) AND F I L ( NODE ,KP ) = "UP " 

DO 

CREATE A MESSAGE 

LET C (MESSAGE )=CI(NODEJ 
LET MtMESSAGE )=NI (NODE) 

LET OPIGIN(N£SSAGE)=NODE 
LET DEST (MESSAGE )=KR 

SCHEDULE A REC S I V E . A . NESSA G 5 GIVEN MESSAGE AND DEST AT 
TIME.V 4 l.C 
LOOP 

DESTROY MESSAGE CALLED COPY 
LET CT ( NO C E ) = 1 

FOR KT = 1 TC N, WITH FIL (NOCE »K T ) = " READY" AND NI(KT) GT 
ZIL(NCDE.KT) 

LET FIL (NGCE ,KT > = ” 

LET NIL (NCC £ ,KT)=0 
RETURN 
END 

EVENT CLT.FUT 

PRINT 1 LINE WITH TINE.V AS FCLLCWS 

EVENT CUT. PUT ENTERED. THE TIME IS ****** 

LIST ATTRIBUTES CF EACH NCDE 
LIST ATTRIBUTES CF EACH SINK 
LIST TFEE, CONNECTIVITY 
LIST FIL.SFIL 

LIST ZIL, DILL, CIL, NIL, SDIL ,SNIL 

RETURN 

END 

EVENT STOP .SINLLAT ICN 
PRINT 1 LINE WITH TINE.V AS FCLLCWS 
TINE TC STCF, T I N E = ***.** 

STGP 

END 
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APPENDIX B 



ALGORITHM FOR NODAL OPERATIONS 



Table I - Variables of the Algorithm of Table III. 



Variable 

Name 


Meaning 


1 

Domain of 
Values 


P i 


preferred neighbor 


nil, 1,2, ... ,K 


d. 

1 


estimated distance from 
SINK 


12 3 

9 ■** 1 ** 9 1 • • • 


D i* 


estimated distance of 
link (i, Z) 


12 3 

• • • 


n . 
1 


current counter number 


0 12 

\J f f 4— f f • • 


mx . 
i 


largest number m 
received by node i 


0 12 

f -i- f y m m • 


CT 


control flag 


0,1 


N ± U) 


last number m received 
from Z after i 
completed last update 
cycle 


nil, 0 ,1,2,... 


D i (iO 


d + d. „ for last d 

IX, 

received from Z 


°°, 1, 2 , . . . 


F i (A) 


status of link (i, Z) 


DOWN , READY , UP 


z ± U) 


synchronization number 
used by i to bring link 
(i, Z) UP 


0, 1,2 , . . . 
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Table II- Messages received by the algorithm of Table 3. 



Message Format 


Meaning 


Domain of Values 


MSG (m, d, l) 


updating message 


m = 0,1,2, . . . 




from l 


d = “,0,1,2,... 






1 = 1,2 , . . . ,K 


FAIL (£) 


failures detected 


l = 1,2 , . . . ,K 




on link (i, £) 




WAKE (il) 


link (i, £) becomes 


l = 1,2, . . . ,K 


1 


operational 




REQ(m) 


request for new 


= 0,1,2, . . . 


j 

| 


update cycle with 






n SINK 


• 
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TABLE III 



ALGORITHM FOR NODE i-FINITE-STATE-MACHINE TRANSITIONS 



T12 Condition 12 MSG(m=mx^, d ^ 00 , 1 = p^) , CT=0 
Comment 12 m n . . 

l 

Action 12 d. <- min D (k) 

l i 

k :F . (k) =UP 

l 

N . (k) =m 

l 

n ± *■ m; 

k s.t. F. (k) = READY if n. > z . (k) , 

l l l ' 

then F i (k) <- UP, N i (k) <- nil; 
transmit MSG(n.,d.) to all k s.t. F.(k)= 

l i l 

UP and k ^ p^; 

CT <- 1 . 

T13 Condition 13 (MSG(l=p^, d= °®, m) or FAIL ( l=p^) ) , 

CT=0 . 

Comment 13 If MSG, then m n^. 

Action 13 d. <-«» 

if MSG, then n^ <- m 

k s.t. F i (k) = READY, if n^z^k) , then 
F i (k) <- UP, N i (k) <- nil; 
transmit MSG(n^,d^) to all k s.t. F^(k)= 
UP and k ^ p^; 

P i *■ nil; 

CT *■ 1. 

T21 Condition 21 k s.t.F^(k) = UP, then N^ (k) =n^=mx^ ; 

k s.t. F^(k)= UP and D^(k) d^; 

if CT = 0, then MSG; 

D i (Pf) ? 00 

Comment 2 1 d^ ^ °°, ^ nil. 

Action 21 Transmit MSG(n_^,d^) to p^; 

p^ <- k* that achieves min D^(k) ; 

k;F i (k)=UP 

k s.t.F i (k) = UP, set N i (k)<- nil 
CT * 1. 
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T22 Conditions 22 
Action 22 
T23 Same as T13 
T23 Condition 32 



Comment 32 
Action 32 



MSG (m=mx^>n^ , d ^ l=p^) , CT=0 

Same as Action 12 . 

k s.t. F^(k) = UP, mx^= N^(k)> n^, 
D i (k) ji °°. 

P i = nil / d i = oo. 

Let k* achieve min D^(k). 
k:F i (k)=UP 
N . (k) =mx. ; 

i i 

Then p^ *■ k*; 

n i ^ mx i'- 
d. «- D. (k*) ; 

i i 

k s.t. F^(k)= READY, if n^>z^(k), then 
F i (k) «- UP, N i (k) «- nil; 
transmit MSG(n^, d^) to all k s.t.F^(k)= 
UP and k f p^; 

CT •*- 1 . 
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TABLE IV 



ALGORITHM FOR AN ARBITRARY NODE i-MESSAGE HANDLER 



For REQ(m) 

if p^ 7 ^ nil, then send REQ(m) to p^. 

For FAIL (1) 

F i (l) -^DOWN; 

CT «- 0 

Execute FINITE-STATE-MACHINE; 

if p^ i- nil, then send REQ(n^) to p^. 

For MSG(m,d,l) 

if F.(l) = READY, then F i ( 1) •*- UP 

(Comment: m> 2 ^( 1 )); 

N i (l)^ m; 

D i (l)^ d + d i]L ; 
mx^ ^-max {m,mx^;}; 

CTO-*- 0 

Execute FINITE-STATE-MACHINE. 

FOR WAKE ( 1) 

(Comment: Assuming F^(l) = DOWN) 

if i and 1 agree to open link (i,l) then: 

z . ( 1 ) -*-max {n . ,n . } 
l i ' l 

F j ( 1) READY; 

N i (l)^ nil; 

if p^ ^ nil, then send REQ(z_^(l)) to p^. 
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TABLE V 



THE ALGORITHM FOR THE SINK 



For REQ(m) 



CT-h 0; 

Execute FINITE-STATE-MACHINE 
For FAIL ( 1) 

F ± ( 1) 4- DOWN; 

CT 4- 0 ; 

Execute FINITE -STATE -MACHINE 
For MSG(m,d,l) 

N. (1) 4 - m; 

l 

CT 4- 0; 

Execute FINITE -STATE -MACHINE 
For WAKE (1) 

(Comment: F^(l) = DOWN) 

if SINK and 1 agree to open link (SINK,1), then 

F. (1) 4 - READY; 

l 

CT 4- 0; 

Execute FINITE-STATE-MACHINE. 

For START 
CT 0 ; 

Execute FINITE-STATE-MACHINE. 

T12 Condition 12 (CT=0) and (REQ (m =? n 

WAKE or START) . 

Action 12 if (REQ or FAIL or WAKE) , then n 



SINK 



) or FAIL or 



n + 1 • 

n SINK ' 



SINK 

k s.t.F i (k)= READY, then 



F^(k) 4 - nil; transmit MSG(n gINK ,0) to 
all k s.t. F ± (k) = UP;CT 4 - 1. 



T21 Condition 21 



Action 21 



T22 Condition 22 



Action 22 



k s.t. F . (k) = UP, then N . (k) = n 



SINK' 



MSG. 



k s.t.F^(k) = UP, then N^(k) 4 - nil. 



CT 1. 

(CT = 0) and (REQ (m = n 
WAKE) 

Same as Action 12 . 



SINK 



) or FAIL or 
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S = State of Node 
T = Allowable Transition 



FIGURE 9. Finite State Machine 
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